Improving Speaker Adaptation by Adjusting the Adaptation Data Set

Transformation based speaker adaptation techniques, such as Maximum Likelihood Linear Regression (MLLR) require a large amount of adaptation data to robustly estimate the transform matrices. In this paper, we present a new adaptation scheme to make use of adaptation data more effectively, which adjusts the adaptation data according to the decoding results on the same adaptation set. The adjustment is at the sentence and word levels, and is based on information extracted from N-best hypotheses. Experiments on the WSJ 20K task show that this method achieved an additional 10% relative word error rate reduction in supervised adaptation and 2% reduction in unsupervised adaptation compared with conventional MLLR approach.